{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = 'all'  # default is ‘last_expr'\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('/Users/siyuyang/Source/repos/GitHub_MSFT/CameraTraps')  # append this repo to PYTHONPATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "from collections import Counter, defaultdict\n",
    "from random import sample\n",
    "import math\n",
    "\n",
    "from tqdm import tqdm\n",
    "from unidecode import unidecode \n",
    "\n",
    "from data_management.megadb.schema import sequences_schema_check\n",
    "from data_management.annotations.add_bounding_boxes_to_megadb import *\n",
    "from data_management.megadb.converters.cct_to_megadb import make_cct_embedded, process_sequences, write_json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example importer\n",
    "\n",
    "This notebook is a template for how new datasets can be formatted for ingestion into the database.\n",
    "\n",
    "The ideal dataset has both **location** and **sequence** information, in addition to any species or bounding box labels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Give the path to a JSON file where output from this script will be written to. You can then take this file to the .Net app for ingestion to the database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_output = '/path/to/sequence/items.json'  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Name of the dataset**\n",
    "\n",
    "All lower-case; use `_` not `-` if needed\n",
    "\n",
    "A `dataset` is a set of image sequences that are located in the same storage container at the same path (there can be sibling folders at that path which are not a part of this dataset). Since blob storage access is mostly container-based, if the access level on this dataset is `internal`, then it should be in a container all by itself or be in a container where all folders are internal datasets. Thus the subset of human or sensitive images from a larger dataset is usually a separate `dataset`, usually named `largedataset_private`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_name = 'new_dataset_name'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 0 - Add an entry to the `datasets` table\n",
    "\n",
    "For format, see https://github.com/agentmorris/MegaDetector/tree/master/data_management/megadb section on `datasets` table.\n",
    "\n",
    "To insert a new entry to the `datasets` table, use `data_management/megadb/query_and_upsert_examples/upsert_items.ipynb` or do that in Azure Portal (not currently working)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Prepare the `sequence` objects to insert into the database\n",
    "\n",
    "Two possibilities..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1a - If you have metadata in COCO Camera Traps (CCT) format already...\n",
    "\n",
    "For a dataset, you probably have one or two JSONs in the CCT format, one containing image-level species labels and another containing bounding box annotations. Here we combine them and embed any annotation items into the image items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# path to the CCT json, or a loaded json object\n",
    "path_to_image_cct = 'path/to/cct.json'  # set to None if not available\n",
    "path_to_bbox_cct = 'path/to/cct_bboxes.json'  # set to None if not available\n",
    "assert not (path_to_image_cct is None and path_to_bbox_cct is None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might like to process the resulting `embedded` dataset a little more:\n",
    "\n",
    "- For image entries that do not have species label but have bounding box annotations, you can add a `species` field to the `annotations` field of each item in the list `embedded`, according to the `category` field of the first `bbox` item in `annotations`:\n",
    "    - If `e['annotations']['bbox'][category']` is `person`, assign `['human']` to `e['annotations']['species']`. Note that it needs to be a list (of one item).\n",
    "    - If `animal`, assign `['unidentified']`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "embedded = make_cct_embedded(image_db=path_to_image_cct, bbox_db=path_to_bbox_cct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following step, properties will be moved to the highest level that is still correct, i.e. if a property at the image-level always has the smae value for all images in a sequence, it will be moved to be a sequence-level property.\n",
    "\n",
    "If a sequence-level property has the same value throughout this dataset (often 'rights holder'), it will be removed from the `sequence` objects. A message about this will be printed, and you should add that property and its (constant) value to this dataset's entry in the `datasets` table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "sequences = process_sequences(embedded, dataset_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# sample some sequences to make sure they are what you expect\n",
    "sample(sequences, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1b - If you're starting from scratch...\n",
    "\n",
    "i.e. starting from metadata in another format, e.g. spreadsheets, CSVs, \n",
    "\n",
    "you would need to compile the metadata into a list of `sequence` items - format specified at https://github.com/agentmorris/MegaDetector/tree/master/data_management/megadb.\n",
    "\n",
    "Call the result `sequences`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Pass the schema check\n",
    "\n",
    "Once your metadata are in the MegaDB format for `sequence` items, we check that they conform to the format's schema.\n",
    "\n",
    "If the format conforms, the following messages will be printed:\n",
    "\n",
    "```\n",
    "Verified that the sequence items meet requirements not captured by the schema.\n",
    "Verified that the sequence items conform to the schema.\n",
    "```\n",
    "\n",
    "For large datasets, the second step will take some time (~ a minute). \n",
    "\n",
    "Otherwise there will be an error message describing what's wrong. Please fix the issues until all checks are passed. You might need to write some snippets of code to loop through the `sequence` items to understand which entries have problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "sequences_schema_check.sequences_schema_check(sequences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Add any iMerit bbox annotations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotation_path = 'path/to/folder_containing_imerit_csvs'  # dataset_name will be the part before `+`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_filename_to_bboxes = extract_annotations(annotation_path, dataset_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequences, images_updated = add_annotations_to_sequences(sequences, image_filename_to_bboxes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sample some sequences to make sure things look good\n",
    "\n",
    "sample(sequences, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Save the `sequence` items to a file\n",
    "\n",
    "You can now take the resulting JSON file to the .Net application for bulk insertion to the database:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(path_to_output, 'w', encoding='utf-8') as f:\n",
    "    json.dump(sequences, f, ensure_ascii=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check that the bounding box annotations and paths to images all survived by running the `visualization/visualize_megadb.py` using the above exported file."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cameratraps",
   "language": "python",
   "name": "conda-env-cameratraps-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
